2020 Linux Kernel 
History Report 


v5.8 A Publication of The Linux Foundation | August 2020 


Signed-off-by: e wart <kstewart@linuxfoundation.org: 

Signed-off-by: $ khan@linuxfoundation.org 

Signed-off-by: Daniel M German <dmg@uvic.cz 

Reviewed-by ah-Hartman <gregkh@linuxfoundation.org 
ed-by: Jon Corb ‘cor net 

Reviewed-by t konstantin@linuxfoundation.org: 

Reviewed-by 7 1eeler@linuxfoundation.org 

Reviewed-by: J  <jperlow@linuxfoundation.org: 

Reviewed-by: S swinslow@linuxfoundation.org: 

Reviewed-b olan <mdolan@linuxfoundation.« 

Reviewed-b ss <ccr@linuxfot ion.c 

Reviewed-b \lison Rowan <arowan@ oundation.org: 


THE 


L LINUX 


FOUNDATION 


wwW.linuxfoundation.org 


Contents 


Summary 

Introduction 

Kernel Archeology 

Impact of Development Process Best Practices 
Adoption of Maintainer Hierarchy 
Version Control Systems 
Removing Unused Code 
Highly Diversified Corporate Contributors 
Tagging and adoption of the Developer Certificate of Origin 
Release Model with Predictable Release Cycle Cadence 
Improving Automated Testing the Kernel 
Stable Release Process 
Longterm Release Kernels 

Conclusion 


References 


With the 5.8 release tagging on August 2, 2020", and 
with the merge window for 5.9 now complete, over a 
million commits of recorded Linux Kernel history are 
available to analyze from the last 29 years. This report 
looks back through the history of the Linux kernel and 
the impact of some of the best practices and tooling 
infrastructure that has emerged to enable one of 

the largest software collaborations known. The 5.8 
kernel set several records’, so there are no signs of 
development slowing down. 
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Introduction 


Over the last few decades, we've seen Linux steadily 
grow and become the most widely used operating 
system kernel. From sensors to supercomputers, we're 
seeing it used for launching rockets, automobiles, 
smartphones, watches, and many more devices in our 
everyday lives. Since the Linux Foundation started 
publishing the Linux Kernel Development Reports in 
2008?, we've observed progress between points in time. 


However, we haven't been able to share the full scope 
of development since that first release in 1991, Since 
that original release, Linux has become one of the 
most successful collaborations in history, with over 20 
thousand contributors over the 29 years, and over a 
million commits as of August 2020. Given the recent 
announcement of 5.8 release as one of the largest yet, 
there's no sign of it slowing down, with the latest release 
showing a new record of over ten commits per hour’. 


In this report, we're going to look at Linux since the first 
release on September 17, 1991, to August 2, 2020, This 
historical analysis is possible thanks to the efforts of 
Dr. Daniel German and his work on the tool cregit**, 
With this tool, we can look back to the first release on 
September 17, 1991°, and see the kernel commits’ history 
at the token level by contributors up to the latest 5.8 
release on August 2, 2020. Using cregit, we have been 
able to connect the three different development stages 


of Linux development: pre-version control, BitKeeper, 
and Git. During the pre-version control stage, we 

use each of the releases of Linux as a snapshot of 

its development. This stage lasted from September 
1991 until February 4, 2002. February 4, 2002, is when 
BitKeeper Started being used and marks the start of 
the commit history for the Linux kernel. The current 
version control system, Git, started being used on 
April 16, 2005. The development histories of these 
three stages were connected, allowing us to track the 
evolution of kernel source code’. 


Git can track the evolution of Linux at the line level. 
Using Git, one can identify the change that introduced 
or last changed any line of code. Using cregit, it was 
possible to track the evolution of the kernel at the 
token level. A token of a source code program is the 
smallest individual element of a program (a token 

in programming is similar to the concept of word in 
human languages). Hence, it was possible to track, for 
every token in each line, who and when introduced it. 


When linux-0.01.tar.Z kernel was released, it 
consisted of 88 files and 10,239 lines of code and ran on 
a single hardware architecture, i386°. Today, the v5.8 
release consists of 69,325 files and 28,442,673 lines of 
code (which corresponds to 110,354,844 tokens), and 
it runs on over 30 major hardware architectures’. 


The Linux Foun 


Kernel 
Archeology 


In Linus’s note about the statistics on that first release®, 
the initial release is summarized as “It's 88 files 
with about ten thousand lines, written by 
yours truly except for the vsprintf routine 
which was co-written with Lars Wirzenius.” 
When looking at the vsprintf file today, it’s one of the 
few files with source code from that first commit. 
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2,964 tokens can be traced back to 1991 that can still be 
found in the 5.8 kernel. In fact, tokens remain in the 5.8 
kernel from every year since then. Table 1 shows the 

breakdown of the year in which tokens were introdu 
that are still in the kern 
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As Table 1 illustrates, over half of the code in the 5.8 
kernel was written in the last seven years, but traces 


exist from all the prior years contribute to the code in 
version 5.8 even today. 
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1997 
1998 
1999 
2000 
2001 
2002 
2003 
2004 
2005 
2006 


2014 
2015 
2016 
2017 
2018 
2019 
2020* 


# of Tokens 


436,941 
775,706 
1,469,450 
828,530 
2,475,002 
1,310,598 
2,023,705 
2,575,195 
2,449,966 


5,461,240 
4,805,525 

9,906 
6,072,494 


7,622,786 
11,028,463 


Proportion 
0.00% 
0.03% 
0.04% 


0.19% 
0.34% 
0.40% 
0.70% 
1.33% 
0.75% 
2.24% 
1.19% 


2.33% 
2.22% 
2.83% 
4.00% 
4.95% 

35% 
5.31% 
5.50% 


Accumulated 


0.0 
0.11% 
0.19% 
0.38% 
0.72% 
1.11% 
1.81% 
3.15% 
3.90% 
6.14% 
7.33% 
9.16% 
11.49% 
13.71% 
16.54% 
20.54% 
25.49% 
29.84% 


40.65% 
46.49% 
52.01% 
60.14% 
67.04% 
77.04% 
85.42% 
95.24% 
100.00% 
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Impact of Development Process 


Best Practices 


In 2015, The Linux Foundation’s Core Infrastructure 
Initiative (CII) program introduced a best practices 
badge for open source projects to publicly determine 
their software development and security practices". 
The Linux kernel was one of the first projects to get a 
badge. As time has evolved, additional practices have 
been identified, and these were incorporated into 
higher badge levels for projects to strive for”. In June 
of this year, the Linux kernel joined the small handful 
of projects with a gold badge, the top badge level'*. 
This is a visible recognition of what's been there for a 
long time now; the Linux kernel community continues 
to lead on establishing best practices in the areas of 
software engineering and secure development at scale. 
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Contributors to the Linux Kernel per Year 


The kernel maintainers meet in person every year 
(except this one) at the Maintainer Summit to discuss 
how to improve their ways of working together. Topics 
for the summit are determined by developers raising 
them on the ksummit-discuss mailing list™. After 

a healthy discussion, those that aren't resolved are 
selected to be discussed in person. These meetings are 
vital to the continuous improvement of the processes 
that the community follows. Over time there has been 
a steady growth in the number of contributors and 
commits to the kernel each year. While it is not easy 
to quantify the impact of these process improvement 
meetings, they are likely a positive factor. 


100,000 
75,000 


50,000 


commits 


25,000 


o 
Too oS Fe eee 
FPF PFS FPP rg se gs 


year 


Commits to the Linux Kernel per Year 


The Linux Foundation 7 


400,000 
300,000 
a 
3 
B 200,000 
3 
$ 
= 
$ 
100,000 
o 
D 
Ki GX 


2020 Linux Kernel History Report 


CHL MP I 
Pf > 


Adoption of Maintainer Hierarchy 


One of the best practices that have helped the kernel 
scale over time is the adoption of a maintainer hierarchy. 
If we look back through the kernel release archives, the 
first instance of the MAINTAINERS file was committed 
as part of v1.3.68 in January 1996". The file was only 
107 lines long, and there were three maintainers listed, 
Alan Cox, Jon Naylor, and Linus Torvalds. That version 
of the MAINTAINERS file ends with: 


REST: 
By Linus Torvalds 
S: Buried alive in email 


In trying to understand the discussions that led to 

the formation of this file, early Linux development 
mailing lists were consulted. Unfortunately, only partial 
records of the discussions are publicly available before 
1997, as Linux development took place across multiple 


Emails on LKML per Year 


mailing lists and USENET groups. Before the linux- 
kernel mailing list was hosted on vger.kernel.org, 

it was at vger. rutgers.edu, which was only one of a 
number of the lists where the discussions occurred, 
with significant discussions still on USENET. One archive 
source is http://1kml.iu.edu/hypermail/linux/ 


kernel/, containing linux-kernel mailing list (“LKML") 


archives going back to 1995, but even it contains some 
key gaps. If anyone reading this has access to early 
Linux discussion threads, please reach out to the lore. 
kernel.org administrators'® so we can add more early 
development discussions to the archives"®. 


In contrast, today’s v5.8 MAINTAINERS file is now 19,033 
lines long and has 1501" maintainers listed'®. The v5.8 
MAINTAINERS file ends with: 


THE REST 

M: Linus Torvalds <torvalds@1linux-foundation.org> 
L: linux-kernel@vger.kernel.org 

S: Buried alive in reporters 

Q: http://patchwork.kernel.org/project /LKML/list/ 
T: git git://git.kernel.org/pub/scm/linux/ 
kernel/git/torvalds/linux.git 

F: * 

Fi */ 


As the number of maintainers has increased, the 
number of email messages on LMKL has increased as 
well, but it is no longer mentioned as a problem by 
Linus, at least. 
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Version 

Control System Commits 
BitKeeper 15,474 
BitKeeper 18,609 


BitKeeper 22,520 


BitKeeper/Git 23,553 
Git 29,256 
33,761 
48,851 


75,650 
75,827 
77,041 
80,826 
80,097 
82,371 


Table 2: Commits and Contributors over time. 


Version Control Systems 


The version control system choice selected has had 

a significant impact on the community's ability to 
collaborate. This was a topic of much frustration in the 
early 2000s and spurred the break from kernel work 
and creation of Git by Linus Torvalds". It's been over 
15 years since Git development began on April 10, 
2005, and it's proven to be very effective not only for 
the kernel community but also for many other projects. 


Before BitKeeper's use, we lacked transparency as to 
who was contributing, the number of contributions, 
and the paths the contributions flowed through. For 
this report, the commit history is considered to have 
started in 2002. Prior to that, we have the release 
history stored at Linux Kernel Historic Tree” and Dave 
Jones’ history.git”' to identify the earlier release 
information. The CREDITS file in v1.0 gives a listing of 
the early contributors, some of whom are still familiar 
names today”. 


There was a significant increase in the number of visible 
commits and the number of contributors occurring 
each year after the initial transition to BitKeeper. Then 
it's been steady growth after the transition to using Git 
for version control. 


Since the version control system was changed from 


BitKeeper to Git, there's been a steady increase in the 
number of contributors and commits each year as well. 
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commit 1da177e4c3£41524e886b7£1b8a0c1fc7321cac2 
Author: Linus Torvalds <torvalds@ppc970.osdl.org> 
Date: Sat Apr 16 15:20:36 2005 -0700 
Linux-2.6.12-re2 


Initial git repository build. I’m not 
bothering with the full history, even 

though we have it. We can create a separate 
“historical” git archive of that later if we 
want to, and in the meantime it’s about 3.2GB 
when imported into git - space that would 
just make the early git days unnecessarily 
complicated, when we don’t have a lot of 


good infrastructure for 


Let it rip! 


When the first kernel report was published in 2008°, 
Figure 3 in that original report showed the changes per 


hour for the releases from 2005 to 2008. The report 
showed an average of 2 commits per hour for the 2.6.12 
release 15 years ago in May of 2005. Fifteen years later, 
the average for 2019 is 9.4 commits per hour in the 
kernel for the last year. With the latest 5.8 kernel, the 
average was 10.7 commits per hour. 


Having the right tool for the task has made a difference! 


While it is not always possible to identify the gender 
of the contributors to the Linux Kernel, the data we 
have suggests a slight improvement in the diversity of 
contributors in recent years. 


At the end of 2019, we see that 8,5% of the contributors 
were estimated to be female. And as a portion of 

the overall contributors, the percentage of female 
contributors has been increasing over time. 


To identify the gender of contributors, we used 
GenderComputer® developed by researchers at the 
Eindhoven University of Technology and Carnegie 
Mellon University. When considered against the total 
contributors to the kernel though, there is still clearly 
room for the kernel for improving the gender diversity 
of participants. 
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2006 


2008 2010 2012 2014 2016 2018 


year 


% of Contributors to Linux Kernel that are Estimated to be Female 


The Linux Kernel community members continue to look 
for ways to improve the diversity of contributors, and 

it has been a topic at past Kernel Summits”. Kernel 
developers volunteer their own time to mentor new 
developers participating in programs like Outreachy”*, 
and the Linux Foundation's Kernel Mentorship(LKMP) 
program. LKMP is part of the mentorship program 

on CommunityBridge’’ which has been helping 
developers new to open source to experiment, learn, 
and contribute to open source communities. It has 
ultimately helped them connect with experts in open 
source communities as well as employers. The program 
is designed to build a healthy ecosystem around the 
open source software we all care about and rely on by 
funding projects, securing code, and connecting with 
talented developers from diverse backgrounds. 


We at the Linux Foundation have also been reaching 
out to new women contributors asking for suggestions 
for improving the materials and guidance available to 
new contributors. A few takeaways from the feedback: 


Online resources are inadequate or outdated and 
require updates. 


More FAQs on subsystems patch submission 
guidelines will be helpful. 


More FAQs or blogs on best practices for 
contributing to Linux Kernel will be helpful. 


Communication about patch acceptance status can 
be improved. 


More beginner-friendly courses and videos such as 
Shuah Khan's LFD103* and Greg Kroah-Hartman’s 
“Write and Submit your first Linux kernel Patch’”? 
are helpful for students with limited resources. 
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Removing Unused Code 


As one would expect, as the number of commits 

has been increasing over time, the kernel has been 
growing in terms of the number of files and lines of 
code. It is important to remember, though, that each 
kernel build only incorporates a fraction of the source 
code's possible files. As Greg Kroah-Hartmann notes 
in his blog “An average laptop uses around 2 million 
lines of kernel code from 5 thousand files to function 
properly, while the Pixel phone uses 3.2 million lines of 
kernel code from 6 thousand files due to the increased 
complexity of a Soc,” 


There is a continual focus to remove unused code 
and files, but as more features and infrastructure 
are added, the number of files continues to increase 
overall, In looking at the files in the kernel over time, 
there are releases (like v4.17 in 2018 where eight 
architectures were removed, and ~180,00 lines were 


80,000 


20,000 


PTT LeeLee 
ie 


Number of Files in the Linux Kernel 


removed*') where significant pruning is done that it can 
be detected in the trend lines. 


In a similar manner to the files, the lines of code that 
make up the kernel have continued to increase. There's 
more than just the source code, though; comments 

are important for understanding as are blank lines for 
keeping it readable. The following table shows the lines 
of code, comment, and blank lines have been calculated 
by cloc”, 


From prior work done by Daniel German, each line 
of code is, on average, about seven tokens, and the 
number of lines of code tracks the total number of 
tokens very closely over time”, 


= otal lines == lines of code == lines of comments (includes blanks) 
30,000,000 


20,000,000 
10,000,000 


1995 2000 2005 2010 2015 2020 


Lines of Code & Comments 
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Highly Diversified Corporate Contributors 


The Linux Kernel has attracted a wide range of 
organizations contributing to it over the years. From 
looking at the commit data from Git, we see that we 
have a very long tail of organizations contributing to 
the kernel. From 2007 to 2019, there were 780,048 


Organization Soorzois s commits accepted into the Linux kernel from 1730 

None 93,225 11.95% organizations. The top 20 organizations listed below 

Intel 78,068 10.01% onea for sisted piet commits; sense a one 
tail of companies that only made a single commit. The 

Red st eae eas top 20 Committers are at left £ 

(Unknown) 31,919 4.09% 

IBM 29,538 3.79% In these tables and graphs, the label of “(Unknown)” 

SUSE 27,239 3.49% are those patches for which a supporting employer's 

Linaro 24,740 3.17% existence could not be determined. When “None” is 

(coment ~ 23,081 2.96% used, it indicates the patches from developers known 

came 21779 ce to be working on their own time. This convention is 
used in the gitdm tool** that was used to generate 

Samsung 20160 LEED these tables and was initially documented in the 

AMD 17,781 2.28% statistics about the 2.6.20 release”, 

Renesas Electronics 15,542 1.99% 

Texas Instruments 13,855 1.78% 

Oracle 13,295 1.70% 

Broadcom 9,572 1.23% 

Huawei Technologies 9,379 1.20% 

Mellanox 9,267 1.19% 

NXP Semiconductors 9,223 1.18% 

Arm 7,646 0.98% 

Linux Foundation 6,109 0.78% 


Table 3: Top 20 Committers 


o 
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 


Number of Companies Committing to Linux per Year 
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For the last ten years, we've seen over 400 organizations 
contributing to the Linux kernel each year. A significant 
number of these may only ever submit one commit. 
Looking at where the percentage of each year’s commits 
come from, long tail accounts for about one-third of all 
the commits (labeled as “Others” in the chart at left). 


Over time, contributions from companies vary based 
on business needs and strategies. Some of the top 20 
contributors had not even contributed to the kernel 
back in 2007. Others who were strong contributors 

in 2007, have since been bought/acquired and are no 
longer participating. The diversity of contributors has 
been a strength and continues to provide resilience to 
the project. 


Lopartne Number ot Comm 


Linux Kernel Commits by Company (cumulative) 
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Tagging and adoption of the Developer Certificate of Origin 


One of the other key process changes in the history 

of the Linux kernel was the standardization of the 
Developer Certificate of Origin (DCO)** in 20047. The 
DCO was added to provide additional legal protections 
to developers and users without adding a significant 
process burden. The DCO significantly improved the 
ability to track paths taken by patches to get into the 
kernel. Together with the transition of the version 
control system to Git, DCO has eased the overhead 
for developers to contribute over the last 15 years. It 
has proved very popular with developers and has since 
been adopted by other open source projects. 


= Commits == Commits with Signed-off-by Tags 
== Commits with Review tags (reviewed-by or acked-by) 
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Linux Kernel Commits with Tags per Year 
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With the standardization of the use of the DCO in 
2004 by the kernel community, almost all commits 
now have a Signed-off-by tag associated with them. 
There has also been a steady increase in visible reviews 
associated with each commit. The addition of tags that 
indicate a review (Reviewed-by and Acked-by) helps 
with tracking those interested in an area and have had 
a chance to comment and flag any concerns. Patches 
may need to go through several cycles of review until 
the relevant code maintainers have accepted the 
changes. This level of scrutiny before changes are 
committed is important to the quality of the kernel. 


Signed-off-by tags have improved our knowledge 

of the paths taken by patches into the kernel. On 
average, there are usually two Signed-off-by tags 
associated with each commit, reflecting the hierarchy 
of maintainers prior to the merged code. The use of a 
tag to indicate a review has occurred (either Reviewed- 
by or Acked-by) has also steadily increased since its 
introduction in 2007, 


However, other tags that were introduced, have not 
been as widely adopted and used as originally hoped. 
Tags do provide scope for developer creativity to be 
observed; some of the more interesting ones that 
can be found when reviewing the git repositories 
include: “Enthusiastically-ack'd-by”, “Thanks- 
to”, “Based-on-the-original-screenplay-by”, 
and “Catched-by-and-rightfully-ranted-at-by”. 
Moving forward, it would be great to see more use of 
“Tested-by”. 
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# Tags of Reviews (reviewed-by or acked-by) 
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Release Model with Predictable Release Cycle Cadence 


The Linux Kernel Release Model has evolved, and 
releases are considered to be classified into one of four 
categories: Prepatch (or “-rc”) kernels, Mainline, Stable, 
and Long Term Stable**. Details and links to the active 
Kernel releases can be found on https://www.kernel.org/. 


The Linux Kernel Archives 


About 


col Location 


Contact us FAQ 


Releases Signatures Sitenews 


https://www.kemeLorg/pub/ 
https://git.kerneLorg/ 


414,193 
49.232 
4.4232 
linux-next: next-20200820 2020-08-20 


rsync://rsync.kerneLorg/pub/ 


2020-08-16 [tarball] [patch] [view diff] [browse] 

2020-08-19 [tarball] [pgp] [patch] [inc. patch] [view diff] [browse] [changelog] 
2020-08-19 [tarball] [pgp] [patch] [inc. patch) [view diff] [browse] [changelog] 
2020-08-19 [tarball] [pgp] [patch] [inc. patch] [view diff] [browse] [changelog] 
2020-08-19 [tarball] [pgp] [patch] [inc. patch] [view diff] [browse] [changelog] 
2020-08-07 [tarball] [pap] [patch] [inc. patch] [view diff] [browse] [changelog] 
2020-07-31 [tarball] [pgp] [patch] [inc. patch] [view diff] [browse] [changelog] 
2020-07-31 [tarball] [pgp] [patch] [inc. patch] [view diff] [browse] [changelog] 
[browse] 


Left: source: https://www,kernel,org/ on 2020-08-20 
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The length of the release cycle cadence has undergone 
much discussion and experimentation by the community. 
Still, since 2011 the community seems to have found 

a release model that works and supports the quality 
they seek. In fact, the release cadence has become so 
predictable, that a “Pointy-Haired-Boss tool”, was 
created so developers could quickly answer their corporate 
managers’ question of “when will it be released?” 


Each release cycle now starts with a two week “merge 
window” when new functionality can be included with the 
appropriate review into the git repository for the upcoming 
release. Once it is tagged rc1, then the cycles of integration 
testing, debugging, and stabilizing commence. A weekly 
rc candidate is then tagged until the target quality and 
stability is reached. The release is made, and the cycle 
commences again with the next merge window. 


More details on the kernel release model and how it 
occurred can be found in Greg Kroah-Hartmann’s blog on 
the history of how the community came up with this model”. 


#ofReleases 


Linux Releases per Year 
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2018 
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Improving Automated 
Testing the Kernel 


Linux Kernel testing is a community effort. Static 
analysis tools such as sparse (Semantic parser)”, 

sma‘ (source matcher)*2, and coccicheck (semantic 
patches that test for specific bugs)*? are run on Linux 
Kernel trees by autotest bots* such as 0-day*°, Hulk 
Robot **, In addition, fuzz testers such as Trinity: Linux 
System Call Fuzzer”, and other tests are run to 
find problems proactively. Fuzz testing*® tools such as 
bot are run on various kernel trees. How do these 
bots and tools fare finding problems? This graph shows 
fixes in the kernel attributed to findings by various 
bots, and tools. 


The Linux kernel 5.4 development release statistics 
highlighted the impact that bots have finding bugs 

and being able to keep up with them®. The use of the 
“Reported-by” tag, attributing the bots, is helping to 


raise awareness of the role automation of testing is 
playing in preventing regressions in each release cycle. 
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Active stable releases come out approximately once 

a week, Let's talk through the release process with 
Linux 5.7.9 as an example. The release candidate 

(rc) is announced in an email to the stable release 
mailing list®° with a pseudo short log of commits 

and information on where to download the release 
candidate from for review and testing”. This release 
candidate has 166 patches and all of these patches get 
sent to the mailing list in separate emails. Individual 
developers and automated test rings (Kernel CI, 
0-day, etc.) test the release candidate and report 
results in replying to the release candidate email. 
Problems found in rci are fixed with subsequent rc2, 
rc3 candidates. Release candidate testing is uneventful 
and releasing rc2 is infrequent. Who does the testing? 


Buildbot (for Linux kernel hwmon and stable- queue) 
builds the release candidate on all 30+ supported 
architectures, including variants such as arm64 and arm 
are built for multiple kernel configurations and reports 
results®. A total of 31 architectures and 56 configs are 
build tested during the stable release review cycle. LKFT 
(Linux Kernel Functional Testing) ring builds and tests 
release candidates on arm and arm64 hardware and 
reports results™. Tests run include Linux Test Project 
(LTP), Linux Kernel Selftests, Linux Perf, and many 
more. For a complete list of tests run in this ring, refer 
to LKFT Test documentation®, and Kernel CI* ring 
runs boot tests on several systems and reports results. 


Once the release candidate gets sufficient testing, it is 
released, and another email is sent out to announce the 
release. 
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Longterm Release Kernels 


The introduction of Longterm release kernels, based 
on stable releases, has further increased Linux's 
popularity with product makers and increased 
adoption in new market segments. Greg Kroah- 
Hartmann wrote an excellent blog on which kernel he'd 
choose for a task in 2018”, and it provides a useful 
overview for developers looking to select a kernel for 
their project today. Details of the supported long term 
stable releases and period until the end of the support 
(Projected EOL) can be found at kernel.org. 


Longterm release kernels start off as a stable kernel 
with a commitment to maintaining for an extended 
period. Distributions like SUSE, Ubuntu, Red Hat, etc. 
helped to pioneer this concept, and it was formalized in 
2010% with the formalization of “longterm” kernels and 


creation of the stable@kernel.org mailing list. 


Bugs found in the current linux stable kernel are fixed 
upstream and then applied to the stable kernel. When 
a fix is determined to be applicable to one of the 

longterm release kernels, it is backported and applied. 


Improvements in tooling to detect when a fix can 
apply to a longterm stable kernel, like the work by 


Released Projected EOL 

2019-11 December 
December 202: 
January 2024 


ha Levin 


January 20 


ha Levin Februar 


Source: https://www.kernel.org/category/releases.html 


Sasha Levin on creating a tool using machine-learning 
techniques to identify patches that look like useful 
fixes*, have been significantly improving in recent 
years, and are now finding more bug fixes that can be 
applied to the longterm release kernels. 


In 2019, 18,668 commits were made to the various 

LTS kernels; that’s over 2.15 commits per hour. More 
changes were made to the LTS releases in 2019 than 
to the 2.6.12 kernel, 15 years ago. 


There is market pressure for even longer support 
windows to support some of the applications 

where Linux is being used. For example, the Civil 
Infrastructure Platform (CIP) project® members have a 
common interest in having an infrastructure that needs 
to be maintained for extended periods. The developers 
participating in this project have decided to support 
4.4 and 4.19 as Super Long-term Stable (SLTS) kernel 
release®', It will be interesting to see what tools and 
processes these developers create to help make this 
vision possible in the years to come. 


20,000 ws 
v4.19 
Bo 
Bs 
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15,000 


10,000 


Commits 


5.000 


Year 


Tools Detect More Fixes for Longterm Linux Kernels 
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In 2020, the Linux kernel earned a gold CII best practices 
badge®?, which demonstrates that the project applies 
many practices to improve quality, improve security, and 
prevent defects. We see the Linux kernel now being 
used in products where security and safety-critical 
considerations are essential, from medical devices™, to 
autonomous vehicles®, and to spacecraft“, Improving 
the infrastructure so that the right level of security and 
safety analysis can be done confidently before using 
Linux in these safety-critical applications is one of the 


next big challenges that is being tackled”, 


The focus of the kernel community to maintain a common 
goal of having a high quality operating system with no 
regressions, willingness to create new processes and 
tools as needed to help them be more efficient continues 
to improve dependability of the Linux kernel as it gets 
deployed in new markets. Tooling improvements emerging 
in the kernel testing infrastructure are helping developers 
keep up with the rate change in the upstream kernel 
and continue to improve the stable kernels and future 
releases as being transparent about the processes 
followed. Kernel developers have demonstrated they 
are willing to question and discuss improvement and 
welcome diverse perspectives. There now exists a great 
foundation for the Linux kernel to continue to lead the 
way in creating best practices that help the entire open 
source software industry. 
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The authors would like to thank the tens of thousands of 
individual kernel contributors; without them, papers like this 


would not be interesting to anyone. 


The authors would also like to thank those developers who 
have take e e effort to create the infrastructure 
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FOUNDATION 


The Linux Foundation promotes, protects and 
standardizes Linux by providing unified resources 
and services needed for open source to successfully 
compete with closed platforms. 


To learn more about The Linux Foundation or our other 
initiatives please visit us at 


